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Abstract 


With the rapid advancement of technology, education is undergoing a transformational change. 
Chatbots have become increasingly popular in recent years and are being utilized as teaching 
assistants to support teachers and students in various ways. However, little research has been 
done on the skills required by teachers to prepare curriculum content using chatbots. The research 
aims to identify the skills teachers need to prepare curriculum content with chatbots. It examines 
the fundamental and cognitive skills individuals need to interpret content generated by chatbots 
and explores the difference between self-assessment and evaluator-based assessment. Fifty-eight 
third-year students, pre-service teachers, in the Elementary education program attempted to 
write a lesson plan using ChatGPT and completed a questionnaire to assess the skills required. 
Their communication with the chatbot as well as their prepared lesson plans were reviewed by an 
evaluator who rated the skills of the participating pre-service teachers. Results indicate that pre- 
service teachers tend to overestimate their skills required to interpret chatbot-generated content 
compared to the evaluator's ratings. Such discrepancies could lead to inaccurate or incomplete 
assessments of their skills, which could hinder their potential for growth and development. 
Keywords: artificial intelligence, chatbots in education, cognitive skills, fundamental skills, 
lessons plan 


Introduction 


In December 2022, the field of artificial intelligence, particularly the area of chatbot 
development and utilization, experienced a surge in growth with the launch of the ChatGPT 
chatbot. However, chatbots are not a new concept, as they have been in existence for 
almost six decades. The first chatbot, called ELIZA, developed by Weizenbaum in 1966, 
utilized pattern matching to simulate psychotherapist conversation with human patients. 
While ELIZA may be unfamiliar to the general public, many individuals are familiar with 
more modern virtual chatbot assistants, such as Amazon's Alexa and Apple's Siri, which 
operate on logical and virtual chatbot systems. The fundamental goal and premise of 
so-called chat-bots is that a computer converses with human clients in natural language 
in a manner that is as human-like as feasible (BradeSko & Mladeni¢é, 2012). They are 
computer programs that mimic and process human communication, allowing people to 
interact with digital devices as if they were speaking with another person (Ciechanowski 
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et al., 2019). The dialogues thereafter typically serve some specific purpose, such as 
conducting web searches, organizing computer files, scheduling appointments, etc., 
depending on the work the bot was designed for (BradeSko & Mladenic¢, 2012). 

Chatbots are increasingly being used in education. Clarizia et al. (2018) introduced 
it as a useful technology for supporting education, as they enable one of the most 
important ways to promote and enable personalized learning, not only increasing support 
and inclusion of students but also significantly reducing the administrative workload 
of teachers and enabling them to focus more on curriculum development and research 
(Cunningham-Nelson et al., 2019). Their advantage is that they are an interactive 
mechanism compared to traditional e-learning systems (Kowalski et at., 2011), and their 
users are only limited by the creativity and imagination of the user (Roos, 2018). In 
2021, Okonkwo and Ade-Ibijola presented a study in which they analyzed 53 articles 
from reputable digital databases, with the aim of understanding the use of chatbots in 
education, including basic information, benefits, challenges, and suggestions for future 
research on the use of chatbots in education. Their study found that chatbot technology is 
used in various areas of education, including teaching and learning (66 %), administration 
(5 %), assessment (6 %), advisory (4 %), and research and development (19 %). They 
highlight that the introduction of chatbot systems in education can bring personalized 
online learning and greater accessibility to learning materials, which students can access 
from anywhere and at any time. Lin and Chang (2020) and Murad et al. (2019) added that 
chatbots are good technological innovations that improve students' interest in learning, 
cognitive skills acquisition, and academic achievements. They also have a successful 
impact on students' motivation (Bii et al., 2018), improve their learning experience 
(Sandu & Gide, 2018) and much more. Georgescu (2018) stated that the effects that 
chatbots can have on education are changing humanity forever with new methods and 
principles. As a complement to existing methods and learning approaches, chatbots 
can play an important role in presenting pedagogical content and assessment. They can 
create new ways of evaluating and providing real-time feedback, among other things. 
In addition to the benefits of using chatbots in education, it is important to mention 
the teacher's skills in their use. Teachers must be skilled in the use of chatbots, which 
means they must have basic skills for working with technology, such as knowledge of 
computer use and internet applications, as well as basic knowledge of programming and 
artificial intelligence. Additionally, it is important for teachers to develop their cognitive 
skills, such as the ability to think critically, adapt their teaching to different learners and 
their needs, and the ability to lead interactions using chatbots, which can become key in 
personalized learning. Teachers must be prepared for changes and adjustments in the use 
of chatbot technology in education and be ready for learning and developing new skills 
that will be required for the effective use of this technology in the classroom. 

Although there are some studies that deal with the area of teacher's use of chatbots 
in teaching (e.g., Bii et al., 2018; Chocarro, 2023), there is a lack of research that 
examines the fundamental and basic cognitive skills that teacher needs for successful and 
productive communication with a chatbot, with the aim of curriculum content planning 
and lesson planning. 
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Research Problem 


The use of chatbots in educational environments requires special skills from 
teachers. Most existing research on the use of chatbots focuses on the opportunities they 
offer teachers and other educators to assess curriculum goals. However, it is difficult to 
find research that explores the skills that teachers need for preparing curriculum content 
with the use of chatbots. They need to understand the concept they want to present to 
their students, they need to be able to design instructional material suitable for use with 
a chatbot, they need to use the correct technologies, and be able to analyze data collected 
from chatbot usage to improve their teaching. To create an effective and interactive 
learning environment with the help of chatbots, teachers need to have certain skills that 
must be defined and monitored to ensure a quality curriculum preparation process with 
the use of chatbots. According to Atlas (2023), chatbots for teachers provide a unique 
opportunity to improve the educational process and connect with students in new ways 
by providing individual feedback and assistance, allowing teachers to focus on other 
tasks. 


Research Focus 


The aim of this study was to explore the fundamental and basic cognitive skills 
that teachers require for preparing curriculum content with the use of chatbots. The 
research focus was based on fundamental research questions: 


RQI: What are fundamental skills that teachers require for interpreting content generated 
by chatbots? 

RQ2: What are basic cognitive skills that teachers require for interpreting content generated 
by chatbots? 

RQ3: How do pre-service teachers assess their fundamental skills required for interpreting 
content generated by chatbots? 

RQ4: How do pre-service teachers assess their own basic cognitive skills required for 
interpreting content generated by chatbots? 

RQ5: What is the difference between self-assessment and evaluator-based assessment 
regarding the fundamental and basic cognitive skills that teachers require for interpreting 
content generated by chatbots? 


Research Methodology 
General Background 


In the academic year 2022/2023, the use of chatbots in the work of students of the 
academic undergraduate program Elementary education, has expanded with the launch 
of the ChatGPT chatbot. They use it as an aid in learning, as a tool for writing essays 
and assignments, and as a means of planning lessons and other curriculum content. 
However, it often becomes apparent that students are not fully proficient in the use of 
chatbots, frequently merely copying generated content without critically evaluating it, 
neither linguistically nor substantively. Additionally, during discussions about their use 
of chatbots, students often overestimate their skills. 
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In order to study the abilities that teachers need to plan the curricular content in 
their teaching, third-year students of the Elementary education undergraduate program 
attempted to write a high-quality and didactically justified lessons plan for teaching 
literature, and engineering and technology in March 2023, using the chatbot ChatGPT 
(5 students who were unable to log into ChatGPT worked with ChatJBT instead). The 
topics varied, but in all cases, they were related to the exact topic that the students had 
recently written a lesson plan, delivered a presentation on, and tested it in the classroom. 
They then led the communication with the chatbot themselves until they were satisfied 
with the plan. Because they had already prepared the plan for the presentation in the 
"traditional way", they were familiar with the theoretical background of the prepared 
curricular content, while also being informed about the appropriate didactic approaches 
to teaching the prepared content. Then they shared the plan with the researcher through 
the 1ka web application, which allows for anonymous submission, and also compared a 
two-part questionnaire to assess the fundamental skills and cognitive abilities necessary 
for the successful use of chatbots for preparing educational content. 

The uniqueness of this study lies in its focus on the skills required by teachers to 
prepare curriculum content using chatbots, a topic that has received little attention in 
previous research. The study explores both fundamental and cognitive skills required 
for interpreting chatbot-generated content, which helps teachers develop the necessary 
competencies to effectively utilize chatbots as teaching aids in their classrooms. 


Sample 


The present study utilized a purposive non-random sampling method consisting 
of 58 third-year students enrolled in the Elementary Education program at the Faculty of 
Education in Slovenia, during the academic year 2022/2023. Of the participants, 12.06 
% were male and 87.94 % were female, and all were aged between 20—22 years. The 
selected participants had previous experience in using the chatbot ChatGPT for academic 
purposes, including writing lesson plans for educational work as part of practical training. 
The sampling procedure aimed to include individuals who were similar to the population 
with respect to the research questions posed. The formation of the sample adhered to 
ethical guidelines, ensuring that participants' rights were not violated, and the treatment 
of results was anonymous. 


Instrument and Procedures 


Prior to conducting the proposed study, a pilot study was carried out. Nine 
individuals (3 students, 3 teachers, 3 university professors) attempted to write a lesson 
plan for teaching literature, and science and technology, using the ChatGPT chatbot. 
Based on their work, the skills required for working with the chatbot were evaluated, and 
fundamental and basic cognitive abilities needed for interpreting the chatbot's outputs 
were identified. Fundamental skills are specific to communicating with a chatbot, while 
basic cognitive skills are more general skills that are useful in interpreting any type of 
text. These abilities were also monitored in the main study. 

The curriculum preparations generated with the chatbots were reviewed by 
a researcher who identified fundamental skills that teachers need to successfully 
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communicate with a chatbot, based on the communication between pre-service teachers 
and the chatbot. The main nine identified skills include: 

1. Ability to recognize and troubleshoot issues: Awareness that the chatbot 
sometimes does not understand the posed question, resulting in an incorrect 
answer. In this case, the individual must identify this problem and find 
appropriate solutions. 

2. Abstract thinking: The ability to think abstractly, recognize different concepts 
related to content, connect ideas, identify patterns, and perform the analytical 
process of thinking. 

3. Awareness that chatbots are capable of learning: Understanding that chatbots 
are programmed to learn and improve their abilities over time, which 
means that the chatbot also learns from its queries, behaviors, and actions, 
contributing to the improvement (or not) of the chatbot's performance. 

4. Creativity: Awareness that chatbots do not necessarily provide appropriate 
solutions or answers to users' questions. In this case, the individual must be 
able to find new ways of communicating with the chatbot. 

5. Incorporating knowledge from diverse fields: The ability to discern and 
recognize the content generated by the chatbot, as it may deal with content 
from different thematic areas, which may result in inadequate answers for the 
particular field. 

6. Language skills of the individual: The ability to express questions and requests 
in a language format that the chatbot understands, usually in the form of clear 
and understandable thoughts. Additionally, the individual must be able to 
understand the chatbot output. 

7. Recognition and understanding of language limitations of the chatbot: The 
ability to recognize and understand the language limitations of chatbots, which 
are programmed to understand only a limited range of linguistic structures 
and expressions. 

8. Understanding concepts: awareness of the context of the generated content, 
understanding it, and understanding the content that the chatbot conveys, 
while also recognizing the purpose of the chatbot and distinguishing between 
different response options. 

9. Understanding the logic and programming of chatbots: Knowledge that 
chatbots are based on certain logic and programming and understanding this 
logic to better understand how the chatbot works and what is expected of it. 


The main seven identified basic cognitive skills for interpreting the text generated by a 
chatbot are: 
1. Attention to detail: An individual must carefully monitor the chatbot's 
responses to ensure that important information is not overlooked. 
2. Comprehension: An individual must be able to understand the language used 
by the chatbot and explain its meaning. 
3. Critical thinking: An individual must be able to evaluate the chatbot's 
responses based on accuracy, relevance, and completeness. 
4. Cultural competence: An individual must be attentive and sensitive to cultural 
differences that may affect or arise in conversations with a chatbot. 


| | https://doi.org/10.33225/BalticSTE/2023.99 


Proceedings of the 5" International Baltic Symposium on Science and Technology Education, BalticSTE2023 | 


104 


5. Emotional intelligence: An individual must be able to recognize and understand 
the emotions conveyed by some chatbots (e.g., Replika, Woebot, Mitsuku, 
Xiaoixe, Cleverbot), and respond appropriately to them. Additionally, 
individuals need emotional intelligence when using a chatbot if questions or 
situations arise that trigger emotional responses. 

6. Logical reasoning: An individual must be able to use logical reasoning skills 
to understand the chatbot's responses and draw conclusions from them. 

7. Memory: An individual must remember the information provided by the 
chatbot in previous conversations. 

8. Problem-solving: At times, an individual must use their problem-solving skills 
to determine how to formulate questions to obtain the necessary information 
from the chatbot. 


Based on the identified fundamental (9) and cognitive (7) skills necessary for 
successful communication with a chatbot, two assessment scales were designed. Within 
the first scale, which focused on fundamental skills, students rated their abilities in abstract 
thinking, language skills, creativity, recognition and understanding of language limitations 
of the chatbot, understanding concepts, understanding the logic and programming of 
chatbots, ability to recognize and troubleshoot issues, incorporating knowledge from 
diverse fields, and awareness that chatbots are capable of learning. The reliability of 
the questionnaire was checked with Cronbach's Alpha coefficient, which is .901 and 
shows excellent internal consistency. Within the second scale, which focused on basic 
cognitive skills, students rated their basic cognitive skills when communicating with a 
chatbot: comprehension, memory, attention to detail, critical thinking, problem-solving, 
logical-reasoning, emotional intelligence, and cultural competence. They assessed the 
abilities on both scales on the basis of a 5-point Likert scale (1 — not capable at all, 2 — not 
capable, 3 — neither capable nor incapable, 4 — capable, 5 — very capable). The reliability 
of the questionnaire was again checked with Cronbach's Alpha coefficient, which is .922 
and also shows excellent internal consistency. Similarly, the researcher reviewed and 
assessed the fundamental and basic cognitive skills of participating students based on 
their submitted reports, re-evaluating the students on a 5-point Likert scale (1 — not 
capable at all, 2 — not capable, 3 — neither capable nor incapable, 4 — capable, 5 — very 
capable). 


Data Analysis 


The internal reliability of the questionnaire was assessed using Cronbach's alpha, 
which confirmed the appropriateness of the questionnaire design. Data from online 
questionnaires were analyzed in terms of descriptive statistics, including the number of 
participants, minimum and maximum response values, mean, and standard deviation. 
The same procedure was used to calculate data related to evaluator ratings. Self- 
evaluation and evaluator-evaluation data were compared to determine the characteristics 
of the relationship between the data. 
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Research Results 


The results were systematically presented according to the addressed skills, 
which were divided into fundamental skills and basic cognitive skills. Table 1 displays 
the outcomes and comparison between self-evaluation and evaluator-evaluation of 
fundamental skills for successful communication with a chatbot that pre-service teachers 
had. 


Table 1 
Self-Evaluation and Evaluator-Evaluation of Fundamental Skills for Successful 
Communication with a Chatbot That Pre-Service Teachers Have 


Grade N Min. Max. M SD Min. Max. M _~ SD x df p 
Student self-evaluation Evaluator-evaluation 

Troubleshoot 572 5 382 .759 1 5 3.37 919 88985 12  .001 
Abstract thinking 58 2 5 3.78 773 1 5 348 1.112 91.814 12  .001 
Chatbotlearning 58 2 5 353 .922 1 5 3.19 963 90160 12  .001 
Creativity 58 = 3 5 393 .697 2 5 3.53 .799 53650 6  .001 
Diverse fields 572 5 391 .714 1 5 340 904 60488 12  .001 
Language skills 58 2 5 405 711 2 5 3.50 884 56801 9  .001 


Chatbot lan- P 


S72 5 3.77 .708 5 3.51 1.020 95.951 12  .001 


guage 
Undersianding: “se. 5 388 781 1 5 344 945 108.284 12 001 
concepts 
aa 571 5 355 1018 1 5 3.09 1.074 95.453 16 001 


Note. Troubleshoot = Ability to recognize and troubleshoot issues; Abstract thinking = Abstract 
thinking; Chatbot learning = Awareness that chatbots are capable of learning; Creativity = Creativity; 
Diverse fields = Incorporating knowledge from diverse fields; Language skills = Language skills of the 
individual; Chatbot language = Recognition and understanding of language limitations of the chatbot; 
Understanding concepts = Understanding concepts; Understanding logic = Understanding the logic and 


programming of chatbots. 


As shown in Table 1, it is evident that pre-service teachers received the highest 
ratings for language skills (/ = 4.05), creativity (VM = 3.92) and incorporating knowledge 
from diverse fields (M = 3.91), while they identified the weakest fundamental skills as 
the ability to recognize and troubleshoot issues, awareness that chatbots are capable 
of learning and understanding the logic and programming of chatbots, recognition and 
understanding of language limitations of the chatbot (VM = 3.77), understanding the 
logic and programming of chatbots (MM = 3.55), awareness that chatbots are capable of 
learning (M = 3.53). The evaluator rated the students highest in creativity (M = 3.53), 
language skills (7 = 3.50) and abstract thinking, and lowest in the ability to recognize 
and troubleshoot issues (/ = 3.37), awareness that chatbots are capable of learning (/ 
= 3.19) in understanding the logic and programming of chatbots (/ = 3.09). In all cases, 
a statistically significant difference between the ratings of participating pre-service 


| | https://doi.org/10.33225/BalticSTE/2023.99 


Proceedings of the 5" International Baltic Symposium on Science and Technology Education, BalticSTE2023 | 


106 


teachers and evaluators was also evident, indicating a statistically significant difference 
in the assessment of students' fundamental skills. The results indicate that students 
rated their fundamental skills for working with a chatbot higher than the evaluator who 
reviewed their products, and the evaluator's response dispersion is higher as well. This 
may suggest that the evaluator had different expectations or criteria for evaluating the 
students' work, or that the students may have overestimated their own abilities. 

In addition, the self-evaluation and evaluator-evaluation of basic cognitive skills 
necessary for successful communication with a chatbot, which are possessed by pre- 
service teachers, were also examined. 


Table 2 
Self-Evaluation and Evaluator-Evaluation of Basic Cognitive Skills for Successful 
Communication with a Chatbot that Pre-Service Teachers Have 


Grade N Min. Max. M SD Min. Max. M SD x df p 
Student self-evaluation Evaluator-evaluation 

Fe renOnAS 5 1)2C«dSO87A 8900S 1Ssi(“‘(‘!SC8«34~SC92B 132.288 16 001 

details 


Comprehension 58 1 5 3.91 844 1 5 357 920 141.546 16 .001 
Critical thinking 58 2 5 3.84 834 1 5 341 992 74105 12 .001 
Cultural compe- 


57 5 349 1.002 1 5 309 1128 -89.617 16 .001 
tence 

Emotionalintelly” gy 4 5 343 1126 1 5 3.28 1.089 986.795 16 .001 
gence 

Logical reasoning 58 2 5 376 802 1 5 342 1.010 69.727 12 .001 
Memory 58 5 374 78 1 5 333 962 102.889 16 .001 
Problem-solving 58 1 5 374 874 1 5 331 1.030 89452 16 .001 


The results presented in Table 2 indicate that pre-service teachers rated their basic 
cognitive skills for communicating with chatbots fairly high. They considered their strong 
areas to be comprehension (M = 3.91), critical thinking (V/ = 3.84) and logical reasoning 
(M = 3.76), while they evaluated their cultural competence (V/ = 3.49) and emotional 
intelligence (M = 3.43) slightly lower. Similarly, the evaluator identified comprehension 
(M = 3.57) and critical thinking (M = 3.41) as strong areas of the participants, while 
the abilities of logical reasoning (VM = 3.12) and cultural competence (MV = 3.09) were 
rated lower. As with fundamental skills, there was also a greater dispersion of evaluator 
responses for basic cognitive skills, but the emotional intelligence of the participants was 
identified as more consistent. Similar to fundamental skills, there was also a statistically 
significant difference between self-evaluation and evaluator rating in basic cognitive 
skills, where again, evaluator ratings were significantly lower than student self-ratings. 


Discussion 
The number of studies exploring the use of chatbots in education is increasing 


(e.g., Kuhail et al., 2023; Yildiz Durak, 2023; Hew et al., 2023). Within the presented 
study, fundamental and basic cognitive skills required by teachers for interpreting content 
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generated by chatbots were identified. The study investigated and presented how pre- 
service teachers assess those skills for interpreting content generated by chatbots. The 
results provide an answer to the question of what the difference is between self-assessment 
and evaluator-based assessment regarding the fundamental and basic cognitive skills that 
teachers require for interpreting content generated by chatbots. When interpreting the 
results, it is necessary to be aware of the challenges, controversies, and opportunities that 
the introduction of chatbots into education brings (Hwang & Chang, 2021; Kooli, 2023). 

The study's results are considered in the context of the research questions. The 
response to the first two research questions defines the fundamental and basic cognitive 
skills that teachers require to interpret content generated by chatbots. The most basic 
fundamental skills that an individual needs for successful communication with chatbots 
include the ability to recognize and troubleshoot issues, abstract thinking, awareness 
that chatbots are capable of learning, creativity, incorporating knowledge from diverse 
fields, language skills of the individual, recognition and understanding of language 
limitations of the chatbot, understanding concepts, and understanding the logic and 
programming of chatbots. On the other hand, basic cognitive skills are more general 
skills that individuals also require for communication with chatbots, including: Attention 
to detail, comprehension, critical thinking, cultural competence, emotional intelligence, 
logical reasoning, memory, and problem solving. 

The main finding of the study is that pre-service teachers assess their fundamental 
skills required for interpreting content generated by chatbots at a higher level than the 
evaluator of their communication skills with the chatbot. Differences emerge that are 
important for the work of pre-service and in-service teachers with chatbots, while also 
considering other factors and variables related to self-evaluation (Mabe & West, 1982). 
Such discrepancies could lead to inaccurate or incomplete assessments of their skills, 
which could have negative consequences in terms of their learning and development. 
Moreover, the statistically significant difference between self-evaluation and evaluator 
rating suggests that the students may not be fully aware of their strengths and weaknesses, 
which could hinder their ability to improve and develop their skills in a meaningful 
way. This is particularly relevant in the context of education, where self-assessment and 
reflection are critical components of the learning process. Inaccurate or inflated self- 
evaluations could prevent students from identifying areas of improvement and taking 
appropriate action to address them, which could limit their potential for growth and 
development. Taking into account all that has been mentioned, we can draw parallels 
with Wollny et al. (2021) — chatbots have the potential to develop into powerful teaching 
tools that can provide insightful feedback to students, but we are not there yet. There is 
still more work to be done in the area of chatbots in education. 

Other aspects and challenges in the research on chatbots in education are also 
significant, as they can greatly influence users' distorted perceptions of chatbots and limit 
their use in education. For instance, Okonkwo in Ade-Ibijola (2021) highlighted ethical 
issues, user relationships, control and maintenance, and recommended well-defined rules 
for chatbot usage that are consistent with user ethics. They also advised conducting more 
research on the functionality of chatbot systems, as it would significantly contribute to a 
positive impact on education. 
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Conclusions and Implications 


The use of chatbots in education is on the rise, with the majority of applications 
focused on teaching and learning, administration, assessment, advisory, and research and 
development. This study discusses the potential of chatbots to revolutionize education by 
making it more accessible, engaging, and personalized. The advantages of using chatbots 
in education include integration of content, quick access, motivation, engagement, 
personalization, and immediate feedback. Nonetheless, there are also challenges to be 
addressed, such as ethical considerations, security issues, and the need for training and 
support for pre-service (and in-service) teachers. Overall, chatbots have great potential 
for improving education and should be further explored and developed in the future. 
This study has implications for the design of educational programs aimed at preparing 
pre-service teachers to work with chatbots. There is a need for more focused training 
on the fundamental skills that pre-service and in-service teachers struggle with. While 
pre-service teachers may have strengths in communicating with chatbots, there are areas 
for improvement, particularly in cultural and emotional intelligence. Overall, the study's 
findings and implications have relevance for teacher training and development programs 
in various countries around the world, as they navigate the potential of chatbots to 
revolutionize education and the challenges that come with their use. 
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